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(54) Selective viewing of video based on one or more themes 



(57) A method for displaying programmatic content 
comprising a first step of indexing within a table seg- 
ments of the programmatic content using at least two 
possibly overlapping thematic categories, then enabling 
user selection of at least one of the thematic categories 



for viewing. The segments of programmatic content are 
arranged into a video sequence responsive to the user- 
selected thematic category. The video sequence is then 
displayed in substantial synch ronicity with annotative in- 
formation associated with a currently viewed segment 
of the video sequence. 
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Description 

BACKGROUND OF THE INVENTION 

[0001] The present Invention relates to a video device i 
for the automatic selective retrieval of possibly non-se- 
quential video segments of a video program, from a sin- 
gle video source, responsive to a viewer's interactive se- 
lection of specific themes inherent in the video source, 
and the display of the selected segments as a seamless n 
video program. 

[0002] As initially conceived, movies and television 
programs were intended to be viewed as linear, sequen- 
tial time experiences, that is, they ran from beginning to 
end, in accordance with the intent of the creator of the is 
piece and at the pacing determined during the editing of 
the work. With the advent of recording devices and per- 
sonal entertainment systems, control over pacing and 
presentation order fell more and more to the viewer. The 
videocassette recorder (VCR) provided primitive func- 20 
tionality including pause, rewind, fast forward and fast 
reverse, thus enabling simple control over the flow of 
time in the experience of the work. However, the level 
of control was necessarily crude and limited. With the 
advent of laser discs, the level of control moved to 25 
frame-accurate cuing, thus increasing the flexibility of 
the viewing experience. However, manual control over 
such detailed cuing was difficult at best. Thus, Bohrman 
(U.S. Patent 5,1 09,482) described a system for compu- 
ter control over a laser disc player that permitted inter- 30 
active selection of frame-accurate clips for selective 
viewing. This system was cumbersome, and required 
the viewer to preview the video to make the indicated 
selections. Thus, Abecassis, in a series of patents (U. 
S. Pat. No. 5,434,678, U.S. Pat. No. 5,589,945, U.S. 35 
Pat. No. 5,664,046, U.S. Pat. No. 5,684,918, U.S. Pat. 
No. 5,69,869, U.S. Pat. No. 5,724,472, U.S. Pat. No. 
5,987,211, U.S. Pat. No. 6,011,895, U.S. Pat. No. 
6,067,401, and U.S. Pat. No. 6,072,934) provided a 
means by which 'experts* could view a video in advance, 40 
and rate each instant of the video along a plurality of 
categories related to the maturity rating of the video, 
such as violence, profanity, bloodshed, nudity, sex, and 
so forth. Then the viewer could define a set of prefer- 
ences for each of these categories, and the system 45 
would automatically select and/or display a subset of the 
original video content that matched those preferences. 
[0003] However, with modem computer technology 
being increasingly applied to television entertainment 
systems, systems exist today for transmitting, receiving, so 
storing, retrieving, and displaying compressed digital 
versions of movies and television programs, with exqui- 
site control over the pacing and ordering of the program 
material. With this increased capability has arisen an in- 
creased desire to personalize the nature of the presen- 55 
tation of entertainment material, and to view and review 
creative works for the purpose of study, analysis and en- 
joyment. The requirements of these latter activities ex- 



tend beyond the simple filtering capabilities envisioned 
and described by Abecassis and Bohrman, and exceed 
the simple censorship analysis described by Von Ko- 
horn in U.S. Patent 4,520,404. 
[0004] An example of a more complex approach to 
this subject is Benson et al. (U.S. Patent 5,574,845), 
who describe a system for analyzing and viewing video 
data based upon models of the video sequence, includ- 
ing time, space, object and event, the event model being 
most similar to the subject of the current invention. In 
the '845 patent, the event model is defined as a se- 
quence of possibly-overlapping episodes, each of which 
is characterized by elements from time and space mod- 
els which also describe the video, and objects from the 
object model of the video. However, this description of 
the video is a strictly structural one, in that the models 
of the video developed in '845 do not take into account 
the syntactic, semantic, or semiotic content or signifi- 
cance of the 'events* depicted in the video. Benson et 
al. describe the use of structural tags to control access 
to and viewing of the video data. 
[0005] What is required is a method and system for 
selectively viewing video content, based upon a existing 
thematic analysis of the content, using interactive selec- 
tion of one or more thematic elements. 

SUMMARY OF THE INVENTION 

[0006] The current invention utilizes interactive selec- 
tion of themes or thematic elements from an audio-vis- 
ual work, to control the content and sequence of the 
viewing of segments of the work. 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0007] 

FIG. 1 is a system diagram for interactive viewing 
of video. 

FIG. 2 is a representation of structural and thematic 
annotation. 

FIG. 3 is an initial screen for viewing an annotated 
work. 

FIG. 4 is a dialog for specifying thematic viewing 
choices. 

FIG. 5 is a schematic illustration of a video and view- 
ing timeline of a portion of the video work selected 
according to teachings of the present invention. 

DETAILED DESCRIPTION 

[0008] The elements of the current system are shown 
generally at 10 in FIG. 1. A control processor reads 
metadata, from a memory device such as memory unit 
12, which describes the structure and content of a film 
or video work. The film or video content is stored in a 
memory device, such as a random access disk or solid- 
state memory unit 14, or may be stored concurrently 
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with the metadata in memory unit 12. The content is 
comprised of a sequence of time-coded video frames 
that are arranged to play in a default order to display the 
entire work. The content and the thematic data need not 
reside on the same physical device, but may be ac- 
cessed via a network or other communication medium. 
[0009] By means of an interactive display 1 6, a control 
processor 1 8 presents to the viewer a series of user in- 
terface control screens by which the user selects one or 
more segments of the video to be viewed. The top level 
selection may be the entire video; but more relevant to 
the current invention is the ability to present a series of 
thematic or structural choices to the user, who can then 
select one or more of these optional views of the work. 
Under interactive control such as through a remote con- 
trol device or other user input device 19, the user can 
then proceed to view the portions of the work appropri- 
ate to the criteria selection, whereby the appropriate vid- 
eo segments are: (1) read from the memory unit, (2) de- 
compressed, and then (3) presented with appropriate 
annotation to the viewer. 

[0010] The structure of the thematic metadata is 
shown schematically in FIG. 2. Throughout the course 
of the work, multiple themes will typically intertwine, so 
that selection of a theme may involve segments of video 
from various portions of the work. When a menu of 
themes is presented to the user, the selections are ex- 
tracted from the metadata file stored in memory 1 2. The 
thematic annotation may be organized in a hierarchy, 
and the user may be afforded the opportunity to select 
an element from one level of the hierarchy, or a multi- 
plicity of elements from a multiplicity of levels of the hi- 
erarchy. Various interface methods common in the art 
may be utilized for this purpose. 
[0011] For any frame in the work, a multiplicity of an- 
notations may apply, including first-order structural ele- 
ments such as scene start or end, presence of an object 
or actor, type of action or content, presence of a song, 
presence of closed captioning information, and so forth. 
Additional higher-order thematic elements may also ap- 
ply, including for example character-specific segments, 
action or musical elements, expository passages, and 
combinations of these elements. These first- and high- 
er-order elements may overlap in general ways. 
[0012] The example in FIG. 2 demonstrates several 
salient characteristics of the annotation elements within 
a video sequence 20 that moves in time from left-to- 
right. Every frame of the work has associated with it at 
least one structural element, and one thematic element. 
The structural elements 22 shown in FIG. 2 are objec- 
tively determinable elements - such as the appearance 
within certain frames of the work of actor 1 , actor 2, a 
song, a red dog, a cedar tree, or an ocean view - whose 
existence within the work are easily determined and can 
be automated by an appropriate apparatus. Thematic 
elements 24 are those subjective elements that drive the 
development of the storyline of the work, such as the 
beginning romance between characters within the work, 



or that portion where jealousy between the characters 
emerges. Although not explicitly shown in FIG. 2, the 
thematic elements may overlap, as where the romance 
portion and jealousy portion begins. For instance, earlier 

5 scenes of the work showing a past boyfriend or girlfriend 
may be appropriate to the jealousy theme as well as the 
romance theme. Structural and thematic elements may 
also overlap in arbitrary ways. 
[0013] FIG. 2 illustrates the example that at time tj, the 

10 structural elements 'actor V, 'song', and 'red dog' exist 
within the video frame time-coded attimet jt and the the- 
matic element 'beginning romance' exists simultaneous 
with the structural elements at that time. Note that the 
themes may exist independent of the objects within the 

is frame so that, for instance, the thematic development of 
the romance between actor 1 and actor 2 may continue 
at time t, despite the non-existence of actor 2 within the 
video frame time-coded at t 1 . Note also that the thematic 
element 'jealousy emerges' does not begin until a later 

20 time-coded sequence of video frames. 

[001 4] When a thematic selection is presented to the 
user, the selection may be accompanied by a keyframe 
taken from the work, This keyframe may be read from 
the compressed video data using means already known 

25 in the art, and then displayed either as an underlay to 
the selection dialog, or as a thumbnail within the selec- 
tion dialog. 

[0015] FIG. 3 shows how the display might look when 
a work is first opened. The opening frame of the movie 

30 js displayed as a still image 26, and two buttons appear 
28, 30 on the bottom of the screen 16. The two buttons 
are part of the button bar control, which at any time dur- 
ing the display of the movie permits the user to step for- 
ward or backward in the time line. At the beginning of 

35 the work, there is no previous scene, so the (previous) 
button normally shown to the left of button 28 is not dis- 
played. The labels in the buttons indicate the content of 
the particular thematic element being displayed, here 
'friends meet' for button 28, and "first argument" for but- 

40 ton 30. 

[001 6] Interaction with the control application may be 
by means of button presses on either a wired or wireless 
remote control, or a wired or wireless keyboard. A pair 
of left/right buttons or a left/right rocker switch on the . 

45 user input means. 1 9 (FIG. 1 ) permits the user to move' 
forward and backward in the timeline of the work. An- 
other key press may initiate an interactive dialog menu 
32, shown in FIG. 4, which permits the user to select 
one or more thematic element to view. 

so [0017] Choices in the top-level thematic dialog win- 
dow 32 may lead to submenus, each of which may pro- 
vide additional or alternative choices, and lead to further 
submenus. For example, selection of 'Actors' at the top 
level may lead to a menu of choices of main characters, 

55 with a selection on that submenu leading to a further 
submenu of choices of minor characters. At each level, 
exclusive or inclusive choices may be made, so that 
combinations of selections may be made which result 
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in combinations of thematic elements being presented. 
This selection feature has three primary embodiments: 
that of union, intersection, and exclusion. Multiple se- 
lections of overlapping portions of the work - whether 
object-theme, theme-theme, object-object, or otherwise 
- may if desired result in the retrieval for viewing from 
memory 1 4 of time-coded video frames associated only 
with the overlapping portions of the selected categories. 
Alternately, multiple category selection may result in the 
retrieval for viewing from memory 14 of time-coded vid- 
eo frames associated with any one of the multiple se- 
lected categories. Finally, inclusive and exclusive 
("NOT 1 ) choices can be made which result in the retriev- 
al and playback of video frames that include certain se- 
lected objects and/or themes and exclude other select- 
ed objects and/or themes. 

[0018] Another aspect of the current invention is the 
display of video and accompanying annotation when 
multiple themes or categories of annotation are select- 
ed. For example, in the case shown in FIG. 4, if 'Actor 

V and 'Actor 2' were both selected for viewing or brows- 
ing, the display sequence may include all those seg- 
ments belonging to either of these objects. The label 
shown during the play of any frame of the video could 
be the label of that selected and visualized segment that 
starts most recently relative to the frame. Here, when 
the 'Actor 1 1 and 'Actor 2' themes are playing concur- 
rently, the label associated with the first 'Actor 2' seg- 
ment would be displayed, until the start of the first 'Actor 

V segment', at which time the label for the first 'Actor 1 ' 
segment would be displayed, having been shown as the 
label of the 'next' button during the display of the first 
'Actor 2' segment. Since the first 'Actor 2' segment con- 
tinues after the first 'Actor 1 ' segment, the label for the 
first 'Actor 2' segment would appear on both the 'previ- 
ous and 'next' buttons during the play of the first 'Actor 
1 ' segment. Once the end of the first 'Actor 1' segment 
was reached, the first 'Actor 2' segment would continue 
to play to its conclusion, with the appropriate label shifts. 
This would be followed by a jump to the second 'Actor 
1' segment. 

[001 9] FIG. 5 illustrates the above sequence with ref- 
erence to the objects and themes shown in FIG. 2. The 
video sequence timeline is shown at 50 and includes 
mapped thereon the time-coded video frames making 
up the video sequence 20. The video sequence shown 
in FIG. 5 includes two video segments 52, 54 shown in 
cross-hatching that do not include therein either Actor 1 
or Actor 2. As these two objects have been selected by 
the user for viewing, all video frames having either Actor 
1 or Actor 2 are retrieved from memory 14 and assem- 
bled for play without interruption as a portion 56 of the 
entire work on playback timeline 58. The solid lines 60, 
62 in video sequence portion 56 denote a non-sequen- 
tial jump in time-coded frames owing to not playing por- 
tions 52 and 54 from the original video sequence 20. 
The dotted lines in both video sequence 20 and portion 
56 denote boundary cues where the selected objects 



and/or themes begin or end an association with frames 
on their respective timelines 50, 58. For instance, dotted 
line 64 denotes the frame within the video sequence 
segment 56 in which Actor 1 first appears on screen with 
s Actor 2, and dotted line 66 denotes the frame in which 
Actor 1 later moves off-screen. 
[0020] Annotative display is responsive to the cue 
frames - such as transition frames 64 and 66 - where 
the content button appearing on the screen just before 
io frame 64 would be reflective of the thematic and object 
selections made. The annotations may be different for 
a particular frame depending upon which combination 
of object and theme elements are selected for viewing 
by the user of the video system. For instance, if Actor 1 
'5 and Actor 2 are selected, then only those annotations 
associated with those objects would appear on the 
screen. Similarly, if a theme is also selected, then the 
appropriate annotations associated with the objects and 
the selected theme are retrieved from memory, such as 
20 from a table stored in metadata memory 12, and dis- 
played on the screen in synchronicity with the display of 
the particular video segments. 
[0021] As an example of the above, filters can be AND 
(union) orOR (intersection) so that thematic annotations 
25 are different depending upon which objects are chosen 
and whether union or intersection is chosen. Selecting 
Actor 1 AND Actor2 would result in displaying allframes 
in the base video sequence that has either Actor 1 in it, 
or Actor 2. Annotative buttons appearing on the screen 
30 with the video payback include text appropriate not only 
to the scene currently played, but also the filter choices 
made. An example of one type of simple annotation is 
by "scene number". Thus, there may be only 7 scenes 
in which Actor 1 and Actor2 both appearand the buttons 
35 may have the numbers "1", "2", "3", etc. displayed on 
them. A more complex set of annotations reflect the re- 
lationship between the objects and/or themes selected 
so that, as shown in FIG. 3, the thematic annotations for 
state "friends meet" and in the next segment "first argu- 
*o ment". 

[0022] In contrast, if the filters included Actor 1 and 
object "Red Dog" from the sequence shown in FIG. 2, 
the annotations may instead be reflective of the relation- 
ship between Actor 1 and the red dog and thus be dif- 
4 5 ferent for a particular video frame shared by both selec- 
tion of actor 1 and actor 2, and of actor 1 and red dog, 
e.g. the video frame with time-coded at time t,. 
[0023] Thematic viewing of a work could be combined 
with a number of control metaphors known in the art, 
50 including for example selectable pull-down lists of the- 
matic elements, permitting random access to segments 
of the work in addition to the sequential access de- 
scribed above. 

[0024] The notion of thematic viewing can applytothe 
55 viewing of multiple distinct video segments related by a 
thematic analysis - that is, thematic viewing can occur 
across multiple works. The second work, like the first 
work, is stored within a memory, such as memory 14, 
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as a second sequence of time-coded video frames ar- 
ranged to play in a default order to display the second 
entire work. Metadata associated with the second work 
are defined and stored as noted above and include the- 
matic categories, at least some of which are in common 
with the thematic categories of the first entire work. The 
portion of the second work associated with the selected 
categories may then be displayed for viewing concur- 
rent with the portion seiected from the first such work. 
Accordingly, for instance, a romance that blossoms be- 
tween two characters in a television series that spans 
multiple episodes can be retrieved from memory and 
strung together to form a seamless display of a new vid- 
eo sequence related only to the romance complete with 
appropriate annotations. 

[0025] The same thematic principles could be applied 
to other works of art or education, including for example 
operatic or orchestral works, writings, poetry, text or 
multimedia training manuals, games, trivia, news broad- 
casts or archives, animation sequences, sporting 
events, disjoint collections of media, or internet search 
results. Again, such thematic principals are not readily 
apparent from the content of the frames themselves but 
rather require expert interpretation of the syntactic, se- 
mantic, or semiotic content or significance of the 'events' 
depicted in the video. 

[0026] Other schemes for segment labeling are pos- 
sible. The developer of the thematic annotation may be 
provided a mechanism for specifying the label to be ap- 
plied at each moment of the video, possibly with context 
sensitivity to the set of thematic elements being shown. 
The label content may depend upon the type and sub- 
type of the segment, may be unique to the particular 
segment, and/or may depend on the other thematic el- 
ements shown at the same time. 
[0027] This method of thematic viewing can be the ba- 
sis for a trivia game, with interaction built into the. view- 
ing process. The interaction may, but not necessarily, 
affect the order in which the appropriate video segments 
are displayed. For instance, if the use is in a trivia game 
with multiple video segments tied together to present a 
complete game, the question order can be presented 
according to some algorithm where the contestants in a 
particular match have answered the questions in one 
particular order but the viewer may want to see the ques- 
tions presented in a different order. The step of display- 
ing the portion of the entire work occurs at least partially 
independent of the time-coded order of the video 
frames." Accordingly the thematic content may be tem- 
porally variable, or may vary according to some algo- 
rithm, thus producing a temporally-varied interactive ex- 
perience. 

[0028] Having described and illustrated the principles 
of the invention in a preferred embodiment thereof, it 
should be apparent that the invention can be modified 
in arrangement and detail without departing from such 
principles. We claim all modifications and variation com- 
ing within the spirit and scope of the following claims. 



Claims 

1 . A computer-implemented method for use by a user 
for management of video data in a stored video 

5 stream, said video stream including a plurality of 
video shots wherein each shot comprises a se- 
quence of frames, said method comprising the 
steps of: 

10 storing within a memory a sequence of time- 

coded video frames arranged to play in a de- 
fault order to display an entire work; 
defining and storing in memory metadata asso- 
ciated with the video frames comprised of a plu- 

15 rality of possibly overlapping thematic catego- 

ries; 

displaying for selection to the user a list of the 
plurality of thematic categories; and 
selecting for viewing a portion of said entire 
20 work associated with the selected thematic cat- 

egory. 

2. The method of claim 1 , further comprising: 

25 correlating the metadata stored in the memory 

with the user-selected thematic category; and 
retrieving for viewing from memory the time- 
coded video frames associated with the user- 
selected thematic category. 

30 

3. The method of claim 1 , further comprising the step 
of displaying the portion of the entire work according 
to the time-coded order of the video frames. 

35 4. The method of claim t , further comprising the step 
of displaying the portion of the entire work at least 
partially independent of the time-coded order of the 
video frames. 

^0 5. The method of claim 1 , further comprising storing 
with the metadata annotations for segments of the 
entire work associated with the content of those 
segments, wherein segments are comprised of a 
plurality of consecutive time-coded video frames. 

45 

6. The method of claim 5, wherein the annotations for 
particular segments are different depending upon 
the seiected thematic category. 

so 7. The method of claim 1 , .further comprising: 

storing within a memory a second sequence of 
time-coded video frames arranged to play in a 
default order to display a second entire work; 
55 defining and storing in memory metadata asso- 

ciated with the second sequence, of video 
frames comprised of a plurality of thematic cat- 
egories in common with said thematic catego- 
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ries of said first entire work; and 
selecting for viewing a portion of said second 
entire work, concurrent with the portion of said 
first entire work, associated with the selected 
thematic category. 5 

8. The method of claim 1 , further comprising the steps 
of selecting two or more thematic categories having 
overlapping portions thereof and retrieving for view- 
ing from memory the time-coded video frames as- 10 
sociated with said overlapping portions. 

9. The method of claim 1 , further comprising the steps 
of selecting two or more thematic categories and 
retrieving for viewing from memory the time-coded is 
video frames associated with any one of said se- 
lected thematic categories. 

10. The method of claim 1, wherein said thematic cat- 
egories at least partially overlap so that a plurality 20 
of video frames are simultaneously associated with 

at least two themes. 

1 1 . A method for displaying programmatic content com- 
prising the steps of: 25 

indexing within a table segments of the pro- 
grammatic content using at least two possibly 
overlapping thematic categories; 
enabling user selection of at least one of the 30 
thematic categories for viewing; 
arranging the segments of programmatic con- 
tent into a video sequence responsive to the us- 
er-selected thematic category; and 
displaying the video sequence in substantial 35 
synch ronicity with annotative information asso- 
ciated with a currently viewed segment of the 
video sequence. 
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